Exhaustive system-level testing of a complex data processing system is unfeasible because there are too many capabilities to test within a reasonable period of time. Therefore, testing is often focused in areas that are likely to be prone to error or critical areas.
A function test is used to determine whether the system correctly performs the function under test. Function tests are useful for verifying correct behavior of the system when performing various functions of the system. However, complex and unpredictable interactions between system functions are difficult to address with function testing because of the number of different functions to be tested.
To cover system behavior not addressed by testing at the function level, the function-level tests are sometimes randomly combined in hopes of creating conditions that test previously uncovered areas of system behavior. Unfortunately, with large systems, the number of possible random combinations and time required to run each test make running enough tests to cover all desired tests areas unfeasible. Thus, test coverage using random combinations of function tests is unpredictable.
If the system-level testing is weighted too heavily toward function testing, complex and unpredictable interactions between components may go untested. Similarly, if the system-level testing is weighted too heavily toward random tests, testing of some critical functions may be overlooked.
A method and apparatus that addresses the aforementioned problems, as well as other related problems, are therefore desirable.